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Abstract 


Exploiting a statewide cutoff point on the placement examination used to assign 
students to remedial courses in Tennessee, this study employs a regression discontinuity 
research design to provide causal estimates of the effects on student outcomes of recently 
redesigned remedial courses at three Tennessee colleges. Moreover, using data on student 
outcomes prior to the course redesigns, the study also tests whether the redesigned remedial 
programs were more effective in preparing students for success in postsecondary education 
than the remedial programs they replaced. The findings indicate that, among students on the 
margins of the cutoff score, the effects of enrollment in developmental mathematics were 
positive and statistically significant on early student persistence as well as on the number of 
credits attempted but not completed in the first semester. However, these effects did not 
persist over time, as the results show no statistically significant differences between groups 
after two years. Yet the study also finds that students who were exposed to redesigned 
developmental math courses had more positive outcomes than did their peers in non- 
redesign institutions during the same period and also when compared with students who 
were exposed to the previous version of traditional remediation within their institution in 
prior years. Students appear to have benefited from redesigned courses at two of the three 
institutions. The results of this analysis provide insight into the extent to which the 
particular instruction and delivery methods of remedial courses affect subsequent student 
academic outcomes, thus informing administrators and policymakers as to how best to help 
underprepared students. 
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1. Introduction 


Large numbers of s tudents who attend college each year are required to enroll in 
remedial programs aimed at enhancing their weak reading, writing, and/or mathematical 
skills to better prepare them for success in college-level courses (Attewell, Lavin, Domina, 
& Levey, 2006; Bailey, Jeong, & Cho, 2010). Recently, a host of new course innovations 
have surfaced that are intended to move students through remediation more efficiently and 
effectively. In Tennessee, the focus of the research described in this paper, several colleges 
have redesigned the way in which they offer remedial courses, including mainstreaming 
students into college-level courses and making greater use of technology to provide 
individualized modules that are tailored to students’ specific academic needs. Yet little 
research has been conducted to estimate the causal effects of course redesigns on student 
academic outcomes and to evaluate how the impact of the new courses compares with that 
of “traditional” remediation. In the current study, I address this gap in the literature by 
evaluating the causal effects on students’ early academic success of recent remedial 
mathematics course redesign efforts implemented at three different colleges in Tennessee. 

Exploiting a statewide cutoff point on the placement examination used to assign 
students to remedial courses, I employ a regression discontinuity research design to provide 
causal estimates of the effects of the redesigned courses on the subsequent academic 
outcomes of students placed in math remediation. Using data on student outcomes prior to 
the course redesign, I also test whether the redesigned remedial programs were more 
effective in preparing such students for success in postsecondary education than were the 
remedial programs they replaced. 

Due to the relatively recent adoption of these reform efforts, I focus on the early 
academic outcomes of students, including persistence from the first to the second semester 
and from the first to the second year, the number of credits attempted but not completed in 
the first semester, and the number of credits (both cumulative and college-level) attained in 
the first two years. I ask the following specific questions: (l)Does participation in 
redesigned remedial courses improve subsequent academic outcomes for s tudents at the 
margins of passing the placement test?; (2) Is participation in redesigned remedial courses 
more effective than participation in traditional remedial courses offered at similar 
institutions during the same time period?; and (3) Is participation in redesigned remedial 
courses more effective than participation in the traditional model of remediation at the same 
institution prior to the implementation of these new courses? 

Findings from this study indicate that, among students on the margins of the cutoff 
score, the effects of enrollment in developmental mathematics were positive and 
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statistically significant on early student persistence as well as on the number of credits 
attempted but not completed in the first semester. Although these effects did not persist 
over time, as I find no statistically significant differences between groups after two years, 
students exposed to redesigned developmental math courses had more positive outcomes 
than did their peers in non-redesign institutions during the same period and also when 
compared with students exposed to the previous version of traditional remediation within 
their institution in prior years. Students appear to have benefited from redesigned courses at 
two of the three institutions. 
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2. Background and Context 

The Problem of Underprepared Students 

Increasing numbers of American students are enrolling in college unprepared for 
college-level work (Greene & Forster, 2003; Attewell et al., 2006; Strong American 
Schools, 2008). In an effort to help these students develop the skills needed to succeed in 
college-level courses, postsecondary institutions offer a range of remedial and 
developmental courses in reading, writing, and mathematics designed to bridge the gap 
between high school and college-level material. Remedial and developmental courses, 
which fall under the general umbrella term of remediation, are designed specifically for 
students with lower level skills or those in need of material below college-level. While the 
terms developmental and remedial are frequently used interchangeably in this literature, 
developmental courses often refer to those courses just below college-level (e.g., Algebra 
II), while remedial courses offer material considerably below college-level (e.g., basic 
arithmetic). 1 

Such courses, and the costs and benefits associated with their delivery, are of 
growing concern to students, taxpayers, and higher-education policymakers at all levels, 
and for good reason: the magnitude and scope of college remediation in the United States is 
immense. Currently, it is estimated that nearly half of all students enrolled in postsecondary 
institutions are in need of a t least one high school-level course (U.S. Department of 
Education, National Center for Education Statistics, 2004; Attewell et ah, 2006; Bailey et 
al., 2010), with some postsecondary institutions reporting that nearly six out of 10 students 
enroll in remedial coursework during their college career (Bettinger & Long, 2009b; Bailey, 

2009) . Statistics from Tennessee show that 73.3 percent of all recent high school graduates 
enrolling in a community college for the first time in the fall of 2010 were in need of at 
least one remedial or developmental course (Tennessee Higher Education Commission, 

20 10) . 2 Within developmental education itself, students are most likely to need help in the 
subject of math (U.S. Department of Education, National Center for Education Statistics, 
2004; Bailey et al., 2010). 


1 In an effort to avoid possible negative connotations associated with the term remedial, practitioners tend 
to use the term developmental education to describe the courses and services offered to students below 
college-level (Bailey et al., 2010). In this paper I refer primarily to developmental mathematics courses as 
those courses just below college-level. Additionally, I occasionally use the term remediation to refer to 
acts or efforts designed to bring students up to college-level courses. 

2 Seventy-three percent is a decrease from 20 years earlier, when 85 percent of all freshman entering 
community colleges in Tennessee required at least one remedial course (Van Allen & Belew, 1992). 
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A recent study estimated the annual cost of remediation at $1.9 to $2.3 billion at 
community colleges and another $500 million at four-year colleges, while several states 
cited costs of tens to hundreds of millions of dollars annually to support remedial programs 
(Strong American Schools, 2008; Collins, 2009). Additionally, students must shoulder the 
tuition costs of the courses. In most postsecondary institutions, remedial and developmental 
courses are typically offered for credit and will count toward a student’s overall GPA, but 
rarely are they counted toward graduation requirements. Yet it is also true that the social 
costs of not offering remediation may be even greater than the institutional costs of the 
programs and the direct costs to students combined. Unskilled individuals incur expenses, 
including unemployment costs, government dependency, and crime (Long, forthcoming, 
2011). As the nation’s economy increasingly demands a more skilled workforce, 
educational institutions are pushed to develop more effective ways to train their workers. 

A growing body of research is emerging on both the scope and effectiveness of 
college remediation. Existing research, however, does not provide clear-cut evidence of the 
benefits of remediation for students. Many previous studies are strictly descriptive in nature 
and simply compare samples of re medial students with their peers, ignoring the fact that 
students in need of re mediation may be different from their more academically prepared 
peers in both their observed and unobserved background characteristics. Comparing these 
two very different types of students while ignoring the problem of unobserved selection can 
lead to biased estimates of the impact of re mediation on subsequent academic outcomes 
(Bettinger & Long, 2009b). Until recently, there has been little research on the causal 
effects of remediation on student outcomes, and existing studies have produced inconsistent 
findings. This research has nonetheless uncovered critical questions about whether remedial 
programs work, on average, to improve student academic outcomes and about which types 
of programs are most effective. Broadly speaking, colleges still know little about the most 
effective ways to provide remedial and developmental courses to improve students’ chances 
for postsecondary success. 


The Effectiveness of Remediation 

The primary purpose of remediation has always been to integrate into college-level 
courses students who may not be ready for c ollege-level work. Two common hypotheses 
have surfaced as to the potential effects of college remediation. On one hand, if remedial 
courses do indeed provide students with the skills they need to be successful academically 
at college and in the labor market afterward, then according to human capital theory, these 


3 Calculating the costs of remediation is likely to become more complicated as postsecondary institutions 
begin to explore alternative methods of offering such courses that do not fit into a traditional, semester- 
long funding formula (Fulton, 2010). 
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courses may be a worthwhile investment of time and resources (Becker, 1993). By helping 
students to develop essential skills, remediation may enable them to succeed in college- 
level courses and persist to graduation more effectively than they otherwise would have 
(Bettinger & Long, 2009b). Additionally, theories of student integration and engagement 
tell us that the additional academic supports offered commonly in remedial courses may 
help to integrate students into their academic environment in important ways, leading to 
higher rates of persistence and completion of their degrees. Students who feel connected to 
their institution (either academically, socially, or both) are more likely to remain enrolled 
than those who feel disconnected (Tinto, 1975; Kuh et ah, 1991; Astin, 1993). If access to, 
and involvement in, college remediation allows a student an opportunity to develop 
confidence in his or her skills, it could also increase student academic engagement and 
improve chances for success indirectly (Astin, 1993; Gray-Barnett, 1999). 

On the other hand, required participation in remedial education may not increase the 
probability that students will succeed in postsecondary education. Discouraging results 
obtained on the impacts of the assessment tests that are used to place students into 
remediation indicate that such tests can cause students to become frustrated, leading 
potentially to increased college drop-out rates (Deil-Amen & Rosenbaum, 2002). 
Furthermore, remedial and developmental courses themselves may slow students in their 
progress toward a degree, given that remedial courses rarely count toward a student’s 
graduation requirements, and given that factors that lengthen the time to degree can reduce 
the probability of degree completion (Bailey, 2009). A recent study by .Jenkins and Cho 
(2012) concluded that students who do not enter a degree program within a year of first 
entering college have a lower probability of eventually earning a degree or credential, 
thereby stressing the importance of s tudents making early progress toward a d egree as an 
important factor in college persistence. Remediation may also have attached stigma, as 
taking remedial courses may lead ultimately to lower self-esteem, higher frustration, and 
higher drop-out rates (Bettinger & Long, 2009a; Jacob & Lefgren, 2004). 

Determining the causal impact of remediation on student outcomes is difficult due 
to the observed and unobserved differences in the students assigned to remediation, as 
compared with students assigned to college-level courses. Simply contrasting the average 
outcomes of these two different groups ignores the problem of selection and tells us nothing 
about whether differences in student outcomes were actually caused by students’ 
enrollment in remedial classes, or whether these differences are instead explained by lower 
levels of academic preparation prior to ever enrolling in remedial courses. Several studies 
have addressed the causal question using a regression discontinuity (RD) design, comparing 
students who were placed into remedial courses by narrowly failing a remediation- 
placement examination with similar students who narrowly passed the same examination 
and then enrolled in college-level courses (Calcagno & Long, 2008; Lesik, 2007; Martorell 
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& McFarlin, 2011). In RD designs, students who score below a specified cutoff score on the 
mandatory remedial placement exam are assigned to a remedial-level course, and students 
scoring above this cutoff are assigned to a college-level course. Assuming that students who 
score just above and below the placement cutoff are equal in expectation prior to treatment, 
one can obtain an unbiased estimate of the causal effect of taking remedial courses on 
subsequent student outcomes for these students at the margins of passing (Shadish, Cook & 
Campbell, 2002; Imbens & Lemieux, 2008). The mixed results from prior research suggest 
that the causal effect of remedial courses on student outcomes is not yet fully understood. 4 5 
Furthermore, most prior research in this area focuses only on evaluating the causal impact 
of participation in traditional, semester-long remedial courses. Recent innovations in 
community colleges and four-year institutions across the country suggest that the traditional 
model of developmental education may be changing. 


Redesigning Developmental Education 

Traditional remedial courses are generally structured in a 15-week, semester-long 
format in which a student takes one remedial course in a given subject before moving on to 
the next course in the sequence. Depending on the student’s prior academic background and 
specific needs, some of these remedial and developmental courses may contain material that 
the student has already mastered. Many are concerned that this traditional model may 
unnecessarily prolong the time to degree and thus increase the probability that students will 
stop out. Moreover, the traditional method of delivering remedial courses often mirrors the 
way students were taught these same subjects in high school. Repeatedly exposing students 
to the same material taught in the same manner may not produce large enough learning 
gains if the instructional format is itself part of the reason for their lack of mastery. 

Redesigning developmental courses can take on a number of purposes and forms. 
Rutschow and Schneider (2011) distilled the multitude of redesign efforts into four types of 
interventions: (a) strategies targeted to students before they enter college, (b) interventions 
that shorten the timing or content of remedial courses, (c) programs that combine basic skill 
attainment with college-level coursework, and (d) supplemental programs such as tutoring, 
advising, or participation in targeted sections outside of class. The process of eliminating 
the developmental courses, which carry no university credit, is referred to commonly as 
mainstreaming. If the method used to assign students to developmental courses is flawed or 


4 New research has also attempted to explore whether the mixed results of prior studies may be explained 
by differences in students’ levels of academic preparation (Bailey et al., 2010; Boatman & Long, 2010). 
This research provides important evidence that suggests that the effects of remediation may differ 
according to individual student need. 

5 For examples of recent evaluations of developmental education interventions, see Howell. Kurlaender, 
and Grodsky (2010) and Jenkins, Zeidenberg, and Kienzl (2009). 
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unreliable, it very well may be that students near the cutoff for assignment to these courses 
would succeed in college-level courses if given the opportunity (Hughes & Scott-Clayton, 
2011). 6 7 Alternatively, if these assessments are accurate in their placement of students into 
remedial courses, then the process of mainstreaming may deny students the opportunity to 
learn material that is only offered in a developmental course. Additional research on other 
types of specific programs suggests that students enrolled in condensed courses, self-paced 
courses, and/or mainstreamed developmental courses do show higher rates of pe rsistence 
subsequently when compared with students taking traditional developmental courses, yet 
causal questions about the effects of these programs on student outcomes remain 
unanswered (Jenkins, Speroni, Belfield, Jaggars, & Edgecombe, 2010; Epper & Baker, 
2009; Zachry, 2008; Edgecombe, 2011). 

In the last several years, a host of states and individual institutions have received 
financial support from government and private sources to provide incentives for redesigning 
and assessing alternative approaches to the ways that they offer remedial and developmental 

H 

education (Couturier, 2011; Carnegie Foundation, 2012; Zachry & Schneider, 2010). An 
increasing number of re design efforts now incorporate the innovative use of learning 
technology into the classroom. These newer models of remediation attempt to better target 
students’ academic needs through improved instructional practice, often through the use of 
learning technology such as self-directed learning labs, online-learning models, and the use 
of high-tech classrooms (Epper & Baker, 2009; Karnjanaprakom, 2012). Such use of 
learning technology could potentially have both positive and negative effects. For example, 
the use of learning technology in the classroom is aimed at shortening the amount of time 
that students spend in developmental courses, thus enabling them to move more quickly 
into their college-level courses while also creating efficiencies in the delivery of 
developmental education. According to the human capital model, by reducing the amount of 
time spent in remediation, the redesigned courses could produce better student outcomes 
due to reducing the direct (tuition) and indirect (foregone earnings) costs of the courses. On 
the other hand, however, the redesigned courses could have no differential impact or could 
even produce more negative effects due to an overreliance on learning technology. Not all 


6 In one study of the Community College of Baltimore County’s Accelerated Learning Program (ALP), 
researchers from the Community College Research Center determined that students participating in a 
mainstreamed English program had higher pass rates in their subsequent college-level courses than did 
their peers who did not enroll in a mainstreamed course (Jenkins et al., 2010). 

7 Six states (Connecticut, Florida, North Carolina, Ohio, Texas, and Virginia) are currently participating 
in the Developmental Education Initiative (DEI), an effort funded by the Bill & Melinda Gates 
Foundation and Lumina Foundation for Education, designed to further advance state policy work in 
developmental education. These states are a subset of the 22 states participating in the Achieving the 
Dream: Community Colleges Count initiative funded by Lumina Foundation for Education and other 
funders. 
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students may be comfortable using learning technology as an instructional tool, particularly 
at the accelerated pace offered in the redesigned courses. 

In this paper, I evaluate three recent efforts to reform developmental math 
instruction at three different colleges in Tennessee. I test the aforementioned hypotheses 
using quasi-experimental methods designed to account for un observed differences among 
the students who are placed into remedial courses versus those who are not. The courses are 
designed to provide students with the skills they need in a more streamlined manner, 
thereby shortening their pathways to college-level courses and increasing their probabilities 
of degree completion (Bailey, 2009). 


The Tennessee Higher Education System and the 
Developmental Course Redesign Initiative 

For the past decade, the Tennessee Board of Regents (TBR) has actively engaged in 
discussions on how to improve remedial and developmental education across its 13 

o 

community colleges and six public universities. In October 2006, the TBR and the 
Education Commission of the States (ECS) received a three-year grant from the Fund for 
the Improvement of P ostsecondary Education (FIPSE) through the U.S. Department of 
Education to implement the Developmental Studies Redesign Project. 9 The project aimed to 
help individual institutions develop and implement a more efficient delivery system for 
remedial and developmental courses with the hope of improving their effectiveness and of 
serving more students better and at less cost (Short, 2009). Starting in the fall 2007, the 
FIPSE funds were distributed to the National Center for Academic Transformation 
(NCAT), a nonprofit organization with expertise in supporting institutions in the use of 
learning technology to improve student learning outcomes. Six proposals out of 1 7 were 
selected as pilot projects designed to last for three semesters, with four proposals focused 
on math and two on English. In the fall of 2007, NCAT awarded pilot grants totaling 
$211,668 to the six selected TBR institutions: Austin Peay State University (math), 


s Tennessee has a higher education system that is similar to most other mid-sized states. The Tennessee 
Higher Education Commission (THEC) coordinates two systems of public higher education in the state: 
the three University of Tennessee institutions governed by the University of Tennessee Board of Trustees, 
and the six state universities and 13 community colleges governed by the Tennessee Board of Regents. 
Together these two systems served over 256,000 students in the fall of 2010 (Tennessee Higher 
Education Commission, 2010). The 19 TBR colleges do not include the five campuses of the University 
of Tennessee system: Knoxville, Chattanooga, Martin, Tullahoma, and Memphis. 

9 For more information on the Developmental Studies Redesign Project, see 
http://tnredesign. 0 rg/ab 0 ut.html# 

10 The selection of the pilot sites was determined by both the quality and feasibility of the proposal. 
Institutions had to demonstrate baseline administrative capacity for making curricular changes and the 
redesign efforts had to be notably different than the existing developmental course structure. 



Cleveland State Community College (math), Jackson State Community College (math), 
Chattanooga State Community College (math), Columbia State Community College 
(reading/writing), and Northeast State Community College (reading). 11 These six 
institutions then took the fall term of 2007 to plan for the implementation of their proposed 
redesign. The technology was installed and tested in the spring of 2008, and full 
implementation of the six pilot sites continued in the fall of 2008 , with four colleges 
reporting successful implementation after the first semester. While all of the institutions 
faced some unanticipated problems during implementation, for Columbia State Community 
College and Chattanooga State Community College, these challenges prohibited their 
ability to implement their redesign plans successfully in the first semester of the pilot year. 
In both cases, instructional and technological aspects of the initial plan were not followed, 
which led to a revision of t he plan midway through the three-semester pilot period 
(Tennessee Developmental Studies Redesign Project, 2008). 

In this paper, I focus on the three institutions that were able to implement reforms in 
their developmental math courses: Austin Peay State University, Cleveland State 
Community College, and Jackson State Community College. Math literacy is perhaps the 
most important need in the nation’s effort to remain competitive in the global economy 
(Epper & Baker, 2009). Yet, of all subject areas, more students enroll in developmental 
math than in any other subject (NCES 2001, as cited in Bettinger & Long, 2009a; Bailey et 
ah, 2010). Furthermore, students are more likely to fail developmental math than any other 
course in higher education (Le, Rogers, & Santos, 2011). Students who fail developmental 
math are also the least likely to ever earn a degree or credential (Le et ah, 2011; Carnegie 
Foundation for the Advancement of Teaching, 2012). 

While the specific details of each institution’s course redesign efforts have differed, 
the overarching goal of a 11 reforms has been to decrease the time students spend in 
developmental math courses. Prior to the Developmental Studies Redesign Initiative, 
developmental courses at the 19 TBR institutions had been taught in much the same way for 
the past 20 years. Courses were offered in traditional 16-week, semester-long formats at 
three levels: basic remedial , basic developmental , and intermediate developmental (Twigg, 
2009). Students in need of remediation were placed into one of these levels for reading, 
writing, and/or math, and were required to complete their assigned course before moving on 


11 NCAT awarded $40,000 in giants each to Austin Peay State University, Jackson State Community 
College, Chattanooga State Community College, and Northeast State Community College. Cleveland 
State Community College received $15,000 and Columbia State Community College received $36,668. 

12 Due to the incomplete implementation of the mathematics course redesign effort at Chattanooga State 
Community College, this institution is not presented in the final analysis. I do, however, present 
institutional characteristics for Chattanooga State in Table 2 and include it in early rounds of the analysis, 
although the results, none of which were statistically significant, are not shown. 
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to the next. Thus, for students in need of multiple remedial courses in the same subject, this 
could mean over a year of course-taking before their remedial requirements were fulfilled. 

The redesigned courses offered innovative structural and instructional changes. 
Chief among these changes was a shift to using learning technology, both in and out of the 
classroom, to enable the students to work at their own pace and to focus their attention 
specifically on t he particular skills in which they were deficient. Each of t he three 
institutions in the pilot redesigned their courses to better tailor the remedial material to the 
students’ specific needs and academic deficiencies. 

The details of each institution’s redesign efforts differed considerably across 
institutions. At Austin Peay State University, both developmental math courses (Algebra I 
and Algebra II) were eliminated entirely, and enhanced sections of the two core college- 
level courses, Fundamentals of Mathematics and Elements of Statistics, were created for 
students whose ACT exam scores placed them in developmental math. These college-level 
courses were linked to Structured Learning Assistance (SLA) workshops in which students 
received additional tutoring and assistance for any course material with which they were 
struggling. Learning technology in the form of computer labs and online tutorials were used 
in the SLA workshops to help bring students up to speed in the college-level material. Due 
to its elimination of the developmental math courses, the Austin Peay model of re form is 
referred to as an example of mainstreaming in the literature. 

Cleveland State Community College adopted an acceleration approach to its 
redesign. Students who completed a developmental math course successfully before the end 
of the term were allowed to begin the next developmental course immediately. Furthermore, 
each developmental course was divided into a smaller number of m odules containing 
subsections of the course material. Students met for one hour in class and for two hours in a 
large computer lab, which allowed them to work online, while instructors provided 
individual student assistance and reviewed student progress. When students completed one 
module, they were allowed to move on to the next, and once they had completed all of the 
modules, they could begin the next course in the developmental sequence. This redesign, 
therefore, required that registration in remedial courses be made more flexible, as students 
were encouraged to complete one developmental course and immediately begin another, 
often in the midst of the semester. 

At Jackson State Community College, all three developmental math courses were 
divided into 12 m odules, with modules 1-3 replacing Basic Arithmetic, modules 4-7 
replacing Developmental Algebra I, and modules 8-12 replacing Developmental Algebra II. 
A pretest was given to each student at the beginning of the semester to determine which 
specific skills students would need to gain for competency in their majors. After the pretest, 
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each student received an individualized learning contract that provided guidance through 
the developmental education pathway. Students were only required to master the concept 
deficiencies determined by the pretest and those that were relevant to their career goals. The 
course content modules were offered in a learning center that also offered video lectures, 
online homework, and weekly testing, as well as immediate assistance from instructors and 
tutors. Students were encouraged to work at their own pace, and weekly assessments 
provided alerts for students who were not grasping the material. For further information, in 
the Appendix I summarize the specific developmental math redesign efforts implemented at 

13 

each institution. 

Observational research and descriptive summaries suggest that these redesigns have 
been highly successful (National Center for Academic Transformation [NCAT], 2009), yet 
research to estimate the causal effects of participating in these redesigned courses on 
subsequent educational outcomes has been notably absent. Descriptively, enrolling in one 
of the three math course redesigns was found to improve subsequent college-level course 
completion rates (as measured by a final grade of C or be tter), as well as to reduce 
instructional costs by 36 percent, on average. At Austin Peay State University, eliminating 
two levels of developmental math and enrolling underprepared students into college-level 
math courses with supplemental instruction resulted in an increase in the overall pass rate of 
underprepared students who required both developmental courses from 17 percent to 76 
percent (NCAT, 2009). At Cleveland State Community College, a study by Schutz and 
Tingle (2010) used logistic regression analysis to determine what effects the course 
redesigns had on student academic outcomes, and the results were similar to those in 
previous studies: strong positive effects were found for next-course success, including the 
next course in both developmental math and college-level math. 

These existing evaluations simply compare the pass rates of students before and 
after the course redesigns were implemented, while failing to account for the selection of 
students into these courses and for any unobserved differences between them and their 
peers who did not take remedial courses. In the research described in this paper, I first use a 
regression discontinuity design to examine how participation in these redesigned courses 
affected the subsequent academic outcomes of target students at the margins of passing the 
placement test. Second, I use the same research design to compare the causal effects of 
enrolling in traditional remedial courses with the effects of enrolling in the newly 
redesigned courses; I do this by comparing the average outcomes of students attending the 
other 16 colleges in Tennessee with those attending “treatment” institutions during the same 
time period. Finally, I compare cohorts of students at the three treatment institutions before 


13 1 include Chattanooga State Community College in the table, although these efforts were not 
implemented fully during the period of the pilot redesigns. 
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and after the curriculum change to further estimate the effects of redesigned courses (versus 
traditional courses) on student academic outcomes. Thus I can determine whether 
enrollment in the recently redesigned courses was more or less effective than enrollment in 
the traditional remedial courses they replaced. 
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3. Research Design 


The manner in which students are assigned to developmental math courses in 
Tennessee provided me with an opportunity to obtain an unbiased estimate of the causal 
effects of enrollment in developmental math courses on students’ subsequent academic 
success. Tennessee is one of several states that use a statewide placement system to assign 
students to remedial courses when they enter college. Since 2005, the primary instrument 
used to assign students to developmental math courses at the public two- and four-year 
colleges has been the ACT Math exam, a subsection of the overall ACT exam. Other 
diagnostic assessments are allowable as secondary or challenge assessments. Community 
college students under the age of 21 without an ACT score are given a placement test prior 
to registering for classes. Entering students 21 years and older who do not have an ACT or 
SAT score also have to take a placement exam. 15 Under the traditional statewide policy in 
place during the years of this study, students with scores between 19-36 points on the ACT 
Math exam are assigned to college-level math, students with scores of 17-18 points are 
assigned to Developmental Algebra II, scores of 15-16 points to Developmental Algebra I, 
and scores below 14 points to Remedial Arithmetic. 

Given that the ACT Math exam is the primary assessment tool used to place 
students into remedial and developmental courses, it is important to examine the differences 
between those who took the ACT and those who did not. In Table 1, I present the sample 
means of select background characteristics and college enrollment information for those 
students who took the ACT Math exam and for those who did not, across the four years of 
2006-07 to 2009-10. Across all 19 TBR institutions over these four cohorts, 82.4 percent of 
all students took the ACT Math exam. Among those students, 98.5 percent were under 21 
years old, with the average age being 18 years old. Eighty-eight percent were enrolled as 
full-time students (registered for a minimum of 12 credit hours per semester) at the start of 
their first year, and these students were enrolled evenly across two- and four-year 
institutions. Conversely, the students who did not take the ACT Math exam (column 3) 
were generally older, with an average age of 2 8 years old, nearly half were enrolled part 


14 Prior to 2005, the state relied on a mathematics placement exam in addition to the ACT to assign 
students to developmental courses. This exam, known as COMPASS (Computerized Adaptive Placement 
Assessment and Support Systems) was administered to students who had a standardized test score (ACT/ 
SAT) below a predetermined threshold or to students who did not take the ACT or SAT. The COMPASS 
exam was then used as the primary took for assigning students to developmental and college-level 
courses. In the fall of 2005, the state began using the ACT as the primary assignment tool. Over 85 
percent of students in Tennessee take the ACT. 

15 These placements exams are most commonly the COMPASS or ASSET (Assessment of Skills for 
Successful Entry and Transfer). Any scores used for initial assessment must have been earned within 
three years prior to the first day of the student’s entering term. 


13 



time, and the majority attended two-year colleges. In this study, I restricted my sample to 
only those students who took the ACT Math exam, which, as I i llustrate in column 2, 
includes primarily students of traditional college age and with full-time enrollment status 
at the start of their first year. 


Table 1 


Comparing Students Who Took the ACT Math Exam with Those Who Did Not 



Full sample 

Took ACT Math 

Did not take 
ACT Math 


(1) 

(2) 

(3) 

Background Characteristics 




Female 

0.565 

0.554 

0.615 

White 

0.706 

0.713 

0.672 

Black 

0.216 

0.210 

0.242 

Other race 

0.057 

0.055 

0.063 

High school GPA 

2.98 

(0.55) 

3.01 

(0.60) 

2.78 

(0.69) 

Average year of high school graduation 

2005 

(4.83) 

2007 

(1.52) 

1999 

(8.22) 

College Enrollment Information 




Age in first semester 

20.12 

(5.25) 

18.23 

(0.90) 

28.37 

(8.25) 

Percent under 21 

0.830 

0.985 

0.102 

Full-time student 

0.822 

0.882 

0.543 

Attend two-year colleges 

0.578 

0.517 

0.860 

Recommend any remedial math course 

0.548 

0.483 

0.850 

Enrolled any remedial math course 

0.448 

0.408 

0.633 

Observations 

111,546 

91,914 

19,632 
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Data 


The Tennessee Higher Education Commission (THEC) and Tennessee Board of 
Regents (TBR) provided the necessary data for this study. THEC and TBR collect basic 
enrollment information and transcript data on e ach student, including courses taken and 
grades for any term during which the student was actively enrolled at any Tennessee public 
institution. Select information is also available on s tudents’ demographic characteristics, 
high school background, and test scores. Key to my analysis, the dataset also includes the 
ACT Math exam scores for all students and a record of their subsequent assignment into 
remedial, developmental, or college-level courses based on this exam. The THEC and TBR 
data to which I have access covers each term (fall, spring, and summer) from the fall of 
2006 through the spring of 201 1. I assigned all students in the dataset to a cohort according 
to the year in which they first began at a public two- or four-year college in the state. By 
using two cohorts before the policy change (students beginning in the fall of 2006 and the 
fall of 2007) and two cohorts after (students beginning in the fall of 2008 and the fall of 
2009), I was able to work with a sample large enough to detect relatively small effect sizes, 
at standard levels of Type I error. 

I also incorporated data from the Integrated Postsecondary Education Data System 
(IPEDS), an annual, federal survey that provides institutional-level characteristics, such as 
the Carnegie classification code, average enrollment, and college graduation rates for each 
institution in the study. These data allowed me to observe potential differences across the 
institutions that may influence student enrollment and course-taking behavior. In Table 2, 1 
show the sample means for student background characteristics and raw enrollment data at 
the institutional level for the four colleges initially receiving FIPSE funds to redesign their 
developmental math curriculum, as well as aggregate measures for all two-year and four- 
year public colleges in the TBR system for the fall of 2008 . Austin Peay State University, 
the only four-year college in the sample, is similar to the other five public four-year TBR 
institutions in the state in terms of the full-time retention rate of first-time students (68 
percent at Austin Peay, compared with 69 percent at the other four-year colleges) and the 
highest percentage of students receiving any financial aid (95 percent at Austin Peay, 
compared with 94 percent at the other four-year colleges). Compared with the five other 
four-year colleges in the state, however, Austin Peay had a higher number of s tudents 
recommended for any remedial math course (45.6 percent versus 35.2 percent, respectively) 
and a lower six-year graduation rate (32 percent versus 41 percent). Among the two-year 
colleges, Cleveland State enrolled a greater percentage of White students (89 percent at 
Cleveland State compared with 77 percent at the other two-year colleges), as well as 
students with slightly higher ACT Math scores and high school GPAs. Cleveland State also 
had the lowest percentage of students recommended for any remedial math course, with 
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Table 2 


Select Institutional Characteristics, Fall 2008 



All TN 
four-year 
colleges 3 

Austin Peay 
State Univ. 

All TN 
two-year 
colleges 11 

Cleveland 
State CC 

Jackson 
State CC 

Chattanooga 
State CC 

Student Characteristics 







Female (%) 

58.3 

(5.1) 

63.1 

62.5 

(5.8) 

61.4 

67.2 

61.7 

Black (%) 

26.0 

(29.3) 

17.2 

14.3 

(17.5) 

5.1 

17.4 

18.8 

White (%) 

65.0 

(27.3) 

62.1 

77.2 

(17.9) 

89.0 

78.3 

78.7 

Age over 25 (%) 

13.2 

(3.7) 

24.4 

16.5 

(5.3) 

21.3 

14.6 

17.7 

Average high school GPA 

3.21 

(0.51) 

3.14 

(0.53) 

2.83 

(0.64) 

2.94 

(0.60) 

2.85 

(0.57) 

2.83 

(0.64) 

Average ACT math score 

20.97 

(4.28) 

20.07 

(3.75) 

17.92 

(3.34) 

18.20 

(3.34) 

18.19 

(3.23) 

17.65 

(3.07) 

ACT math 75th percentile 
score 

22.0 

(2.77) 

23.0 

19.0 

(1.62) 

20.0 

19.0 

19.0 

Student Enrollment Information 







Full-time (FT) C equivalent 
undergraduate enrollment 
Percent of all undergraduates 
beginning as first-time, full- 
time degree- seeking students 

51,821 

25.2 

(13.5) 

7,772 

28.7 

31,165 

52.4 

(11.2) 

2,213 

42.4 

2,900 

44.3 

5,423 

42.8 

Institutional Characteristics 







FT undergraduates receiving 
any financial aid (%) 

94.0 

(2.49) 

95.0 

78.0 

(5.44) 

81.0 

77.0 

79.0 

Recommended for any 
remedial math course within 
the 1st year 

35.2 

45.6 

68.0 

66.8 

70.1 

74.5 

Enrolled in any remedial math 
course in the 1 st year 

24.4 

19.3 

56.7 

60.8 

65.5 

61.1 

FT retention rate d (students 
enrolled fall 2008 returning 
fall 2009) (%) 

69.0 

(5.2) 

68.0 

52.0 

(3.6) 

56.0 

49.0 

53.0 

Received any degree within 6 
years (%) 

41.0 

(4.6) 

32.0 

12.0 

(4.5) 

13.0 

8.0 

8.0 


SOURCE: Integrated Postsecondary Education Data System (IPEDS), NCES, and author’s calculations using THEC 
and TBR institution-level data. 


NOTES: Standard deviations, when available, are shown in parentheses. All data reported for the fall term of 2008-09. 
a The four-year colleges column does not include Austin Peay State University. 

b The two-year colleges column does not include Cleveland State Community College, Jackson State Community 
College, or Chattanooga State Community College. 

c Full-time (FT) students are those enrolled for 12 or more credits per semester. 

d The full-time retention rate applies to students enrolled full time in the fall of 2008 who subsequently enroll foil 
time in the fall of 2009. 
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66.8 percent compared with 70.1 percent at Jackson State and 74.5 percent at Chattanooga 
State. Jackson State and Chattanooga State more closely resembled the other 1 1 two-year 
colleges in Tennessee on race/ethnicity and age characteristics. Both institutions had a low 
six-year graduation rate at 8 percent of the entering cohort, although the average among 
two-year institutions was not much higher at 12 percent. 


Sample 

The sample used in this study contains those students in Tennessee who attended 
one oft he three institutions that implemented a math course redesign successfully 
(described above and in the Appendix) and who also took the ACT Math exam. When 
pooled across all four cohorts (two pre-2008 and two post-2008) among these three 
institutions, the sample size is 8,948 students. 16 Additionally, the sample only includes 
students who began as full-time students so that I am better able to compare credit 
accumulation throughout the early years of college. I define full time as taking a minimum 
of 12 credit hours in the entering term. The vast majority of students in the sample (89 
percent) began as full-time students, making this a weak restriction. I also limit the sample 
to students under the age of 2 1 in order to isolate the effects to traditional students, and only 
include those students for whom I have complete information on gender, race, high school 
grade point average, and postsecondary institution enrollment information. 

In Table 3, I report the sample means and standard deviations for student 
characteristics, including gender, race, high school GPA, and age, as well as the means for 
college-enrollment variables for both the pre-reform (2006-07 and 2007-08) and post- 
reform (2008-09 and 2009-10) cohorts. In doing so, I provide descriptive statistics of the 
sample, and also check that the observable characteristics of students enrolled at the three 
sample institutions pre-2008 were similar to the characteristics of students enrolled post- 
2008 in order to rule out large changes in the composition of the student body during the 
four years of this study. Inferential statistics from accompanying /-tests indicate the extent 
to which the sample means of each variable differ, in the population, between the pre- and 
post-reform cohorts. On average, more female and Black students were recommended for 
developmental math than for college-level math (as shown by rows 1 and 3). Not 
surprisingly, students recommended for college-level math had higher high school grade 
point averages (GPAs) on average than did students recommended for Developmental 
Algebra II, as well as higher average ACT Composite and Math exam scores. Across all 
four cohorts, very few students enrolled in Developmental Algebra II w ho were not 


16 This sample size provides sufficient statistical power (0.90) to detect small treatment effects (0.1 
standard deviations) at the usual levels of Type I error, while still remaining chronologically close to the 
dates of the policy change. 
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recommended for the course, and over 80 percent of those recommended actually did enroll 
in the first semester. 


Table 3 


Sample Characteristics by Level of Course and Time Period of First Enrollment 



RECOMMENDED 


RECOMMENDED 



DEVELOPMENTAL ALGEBRA II 

COLLEGE-LEVEL 



Pre-reform 

Post-reform 


Pre-reform 

Post-reform 



(2006-07 & 

(2008-09 & 

/-test 

(2006-07 & 

(2008-09 & 

/-test 


2007-08) 

2009-10) 


2007-08) 

2009-10) 


Female 

0.621 

0.653 

1.504 

0.523 

0.588 

1.998 

White 

0.725 

0.722 

1.636 

0.767 

0.760 

0.011 

Black 

0.156 

0.162 

0.369 

0.099 

0.093 

0.446 

Other race 

0.089 

0.120 

1.961 

0.107 

0.118 

0.663 

High school GPA 

2.92 

(0.50) 

2.96 

(0.49) 

1.686 

3.08 

(0.50) 

3.15 

(0.48) 

2.660 

Age in 1st semester 

18.07 

(0.56) 

18.13 

(0.55) 

2.167 

18.05 

(0.47) 

18.09 

(0.46) 

1.649 

ACT composite score 

19.32 

(2.12) 

19.41 

(2.11) 

0.964 

20.62 

(1.98) 

21.13 

(1.96) 

5.263 

ACT math score 

17.44 

(0.49) 

17.49 

(0.61) 

2.000 

20.12 

(1.14) 

20.18 

(1.14) 

1.032 

Enroll in any developmental 
math course 

0.866 

0.740 

7.104 

0.004 

0.002 

0.006 

Total number of 

4.28 

(3.42) 

3.16 

(3.73) 


1.02 

(2.07) 

0.89 

(1.77) 


developmental credit hours 
in 1 st semester 11 

6.928 

1.980 

College credit hours in 1st 

9.59 

10.18 

3.217 

13.34 

13.20 

1.054 

semester 

(4.11) 

(4.26) 

(2.82) 

(2.39) 

Observations 

945 

1014 


741 

921 



NOTES: The sample is limited to students who began at Cleveland State Community College, Jackson State 
Community College, or Austin Peay State University in the fall of 2006-07 to 2009-10 with complete infonnation on 
gender, race, age, high school grade point average, and postsecondary institution enrollment information. 


The sample is also limited to students under the age of 21 who began full time. The bandwidth on either side of 
the cutoff (—2 < x < 3 points) was chosen to closely resemble the optimal bandwidths used in the statistical analysis 
of Tables 5-7. 

a The total number of developmental credit hours accumulated in the first semester includes credit hours for 
developmental math, reading, writing, and/or study skills. 
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Among those students recommended for Developmental Algebra II, fewer students 
enrolled in any developmental math course post-reform than pre-reform (74 percent pre- 
reform compared with 86.6 percent post-reform). This difference is likely driven by the 
reform efforts implemented by Austin Peay State University, which eliminated their upper 
two developmental math courses as part of t heir redesign efforts. Therefore, students 
assigned to these courses instead enrolled in college-level math. This policy change also 
explains the observed differences across the pre- and post-cohorts in the number of 
developmental credit hours and the number of c ollege-credit hours attained in the first 
semester. Among students recommended for college-level courses, I only observed 
statistically significant differences between the pre- and post-reform cohorts in the average 
high school GPA and ACT composite score. Both measures increased in the latter two years 
for students recommended to college-level math, although the increases are substantially 
quite small (0.07 GPA points and 0.51 ACT points). 


Measures 

Due to the relatively recent implementation of the redesigns, I measured subsequent 
academic outcomes for students over the short term. Focusing on outcomes in the first two 
years of college allowed me to explore a critical period in the persistence trajectory of an 
undergraduate, as academic performance in the early years is highly predictive of future 
academic success (Bettinger, 2004; Adelman, 2006). Of particular interest in evaluations of 
college remediation efforts is whether assignment to remedial courses slows students down 
in their early progress toward a degree so much so that they become discouraged and stop 
out of college. Early persistence in this paper refers to the probability of remaining enrolled 
in college after the first semester or first year. Given that the sample includes only those 
students who began full time (enrolling for 12 or more credits in their first semester), I 
included in my analysis outcomes for any enrollment in the second semester and year, not 
conditional on prior enrollment. 

I also explored the impact that enrollment in developmental math had on the 
number of both cumulative credits and college-level credits a student had accumulated 
during the first and second year. In recent research, Jenkins and Cho (2012) found it more 
probable that those students who completed nine semester credits, or approximately three 
college courses, in a specific program of study will earn a college credential. This measure 
of “early momentum” in postsecondary education can help predict future success in their 
postsecondary education (Jenkins & Cho, 2012). While the number of total credits in the 
second year may be a good indication of student progress toward a degree, it is the number 


17 1 define persistence as occurring within the same institution in which a student first enrolled. Future 
work will examine transfer students and enrollment in other TBR institutions. 
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of college-level credits completed over time that is most critical to degree attainment. 
Remedial courses in Tennessee count toward a student’s cumulative number of credits, but 
because college-level credit accumulation is the most direct path toward degree completion, 
these are perhaps more important outcomes to consider in the early college years. As such, I 
included as outcome measures the total number of c ollege-level credits completed in the 
second semester, in the second year, and over the first two years. I intentionally excluded 
the first semester when examining the number of c ollege-level credits completed, as we 
would naturally expect to see students assigned to developmental math taking fewer 
college-level credits in the first semester. For students who dropped out, both total credits 
and college credits were imputed as the number of credits when last enrolled. Finally, I also 
included as an outcome a measure of the number of credits a student attempted in the first 
semester but did not complete successfully. This outcome serves as an early measure of 
potential student academic difficulty, as students are generally expected to successfully 
complete all of the courses they begin. 

My principal question predictor was a dichotomous variable that indicates take-up 
of the assignment, or w hether a student actually enrolled in a redesigned remedial or 
developmental course (1 = enrolled, 0 = otherwise) within the first semester. In Tennessee, 
students are encouraged to begin their remedial courses immediately upon enrollment, 
although they are not required to do so. Students must have completed their remedial course 
before they can enroll in the subsequent college-level course, however, which results in the 
majority of students enrolling in their assigned developmental math course within the first 
semester. The majority of students recommended to remedial courses actually enrolled in 
these courses within the first semester (over 80 percent did so). In my analysis, I also 
included a dichotomous predictor that indicates whether the student was enrolled after the 
course redesigns took effect. In addition, my forcing variable was a continuous measure of a 
student’s score on the ACT Math exam centered on the cutoff, with scores of 19 and above 
placing a student into college-level math and scores of 17-18 placing a student into 
developmental math. I centered its values on t he requisite cutoff scores that designate 
assignment to remediation. 

To estimate the impact of the take-up of remediation on the academic outcomes 
(rather than the impact of the offer of remediation), I used assignment to a developmental 
math course at one of the three redesign schools as an instrumental variable in my first- 


18 However, it may be that those who delayed enrollment in their remedial courses to a later semester 
differ in unobserved ways from those who enrolled in their first semester. To check this, 1 conducted 
sensitivity analyses, comparing the results only for students who enrolled in their assigned remedial 
course in their first semester with the results for students who enrolled in a subsequent semester. The 
results were not statistically different from the results for those who enrolled in remedial education in 
their first semester. 
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stage analyses for e ach of my research questions. This dichotomous predictor took on a 
value of one if students scored a 17 or 18 on t he ACT Math exam (the forcing variable), 
indicating that they would be assigned to a remedial course, and zero otherwise. The 
assignment policy is a good choice for an instrument, as it was strongly correlated with 
enrollment in remedial courses, the potentially endogenous question predictor, but was 
determined exogenously by state policy (Calcagno & Long, 2008; Bloom, 2009). 


Analytic Strategy 

All students in this study were assigned to remedial and developmental courses 
using the placement of their ACT Math score with respect to an exogenously defined 
statewide cutoff. Thus, I was able to use a regression discontinuity approach to compare the 
subsequent academic outcomes of students at the margins of passing — those near the 
cutoff who were assigned to the redesigned courses and those near the cutoff who were not 
assigned to the redesigned courses — and thereby obtain an unbiased estimate of the causal 
effect of enrollment in these courses for students at the cutoff (Shadish et ah, 2002; 
DesJardins & McCall, 2007; Mumane & Willett, 2011). My causal inferences assume that, 
other than placement into a higher or lower level course, students immediately on e ither 
side of the cutoff were equal in expectation (i.e., the same, on average, in the population in 
all other respects, both observed and unobserved) prior to treatment. 

However, due to imperfect compliance with the statewide cutoff policy, this 
discontinuity in assignment to remedial classes was fuzzy, in that some students who were 
assigned to remediation did not receive it, and some who were not assigned subsequently 
enrolled in remedial courses. Consequently, I used an instrumental variables (IV) strategy 
and a two-stage least-squares (2SLS) estimation to resolve the fuzziness (Mumane & 
Willett, 2011; Shadish et ah, 2002). I treated assignment to remediation at the cutoff on the 
placement score forcing variable as my instrument for t he potentially endogenous 
enrollment, or take-up, in these remedial courses, and thus obtained an unbiased estimate of 
the causal impact of actual enrollment in a remedial class on subsequent student outcomes. 

In Figure 1, I illustrate the fuzziness in the placement policy. Across all 19 public 
institutions over the four years between 2006-07 and 2009-10, as well as at each of the 
individual institutions in the sample, all students scoring above a 1 9 on the ACT Math exam 
were not recommended for developmental math, but not 1 00 percent of the students scoring 
below a 19 were. 19 The degree of fuzziness differed across each of the three institutions in 


19 While all institutions are supposed to assign students to developmental courses based on students’ 
scores on the ACT Math exam, discussions with officials in Tennessee indicate that exceptions to this 
assignment policy are sometimes made on an individual basis, most commonly as a result of particularly 
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the sample as well. Austin Peay demonstrated the greatest deviation from the placement 
policy, although for students below the cutoff, the assignment rate was still over 85 percent. 


Figure 1 

Percent of Students Assigned to Developmental or Remedial Math by ACT Math Score, 

2006-07 to 2009-10 



Austin Peay State University 



Cleveland State Community College 


Jackson State Community College 




NOTES: All public institutions includes all 19 public colleges in the Tennessee Board of Regents system 
from 2006-07 to 2009-10. Each circle represents the percent of students placed into a developmental or 
remedial math course. The size of the circle represents the relative number of students reporting an ACT 
Math score in the fall of their first year. The vertical lines are drawn at an ACT Math score of 19, the 
statewide cutoff for placement into college-level math. 


In Figure 2, I present the actual enrollment in developmental math versus ACT 
Math score, or the take-up of assignment across all four years in the study. Across the three 


strong high school grades or special student circumstances. These exceptions do not appear to be uniform 
across institutions, nor do they appear to be particularly common. 
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institutions in the sample, on average, over 80 percent of all students assigned to a 
developmental math course enrolled in one of these courses within the first year. In the case 
of the two community colleges in the sample, the enrollment distribution looks similar to 
the assignment distribution shown in Figure 1. For Austin Peay, however, the actual take-up 
of assignment was much lower, at an average of about 60 percent. While this may suggest 
that students were less likely to comply with their assignment after the course redesigns 
took effect, this difference is in part explained by the structure of the redesign itself. Austin 
Peay State University eliminated Developmental Algebra I a nd II and instead enrolled 
students with ACT Math scores below the cutoff into one of two core college-level courses, 
Fundamentals of Math or Elements of Statistics, depending on which course would count 
toward a student’s expected major. As a result, it may appear that fewer students enrolled in 
a developmental math course after 2008, but much of this difference was being driven by an 
elimination of the developmental math course options at this institution. 


Figure 2 

Actual Enrollment in Developmental or Remedial Math by ACT Math Score, 2006-07 to 

2009-10 



Cleveland State Community College Jackson State Community College 
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Research Question 1: Does participation in redesigned remedial courses improve 
subsequent academic outcomes for students at the margins of passing the 
placement test? 

To address my first research question, I restricted my sample to only those students 
attending institutions in which remedial course redesign occurred in the fall of 2008 and 
2009 in order to estimate the causal effect of enrollment in a redesigned remedial course 
(the treatment group) ve rsus no enrollment in these courses (and subsequent unhindered 
enrollment in college-level college courses, the control group). Then, in the first stage of 
my instrumental variables estimation, I fitted the following hypothesized linear probability 
model (Angrist & Pischke, 2009) in which I regressed whether a student enrolled in a 
remedial course on whether a student was assigned to this course based on his/her ACT 
Math exam score centered at the cutoff as follows: 

(1) DEV, = y o + ASSIGN., + y 2 SCORE, + y 3 ( ASSIGN * SCORE ) +;//, + S, 


for the I th individual and where 8\ is the first-stage residual. Z includes exogenous covariates 
describing student gender, race, age, high school GPA, and whether a student had also been 
assigned to a developmental/remedial reading or writing course.” I then introduced the 
fitted probability of being enrolled in developmental courses (from fitting the model 
specified in Equation 1) as the critical question predictor in the following second-stage 
statistical model to estimate the causal effect of the take-up of remediation on outcome, Y h 
as follows: 


(2) Y t =fi o+fl 


DEV, 


+ p 2 (SCORE , ) + P 3 ( ASSIGN * SCORE ) + P 4 (z,. ) + e, 


with a suitable adjustment to the standard errors of the estimates, and where 8; is the 
second-stage residual. The estimates I obtained from fitting the model in Equation 2 
provided the Local Average Treatment Effect (LATE) of the new remediation approach and 
captured only the variation in the outcome that was affected by take-up of the offer of 
developmental math. These results do not provide any information about students whose 
enrollment or non-enrollment in remediation was not influenced by the ACT Math exam 
used for developmental course assignment. Furthermore, these estimates apply only locally 
to the cutoff, and do not apply to students with scores further away from the cutoff than is 
designated by the optimal bandwidth (explained below). 


20 Across all 19 public institutions in the state in the fall of 2008, among those recommended for any 
developmental math course, 42.2 percent were also recommended for a developmental reading course and 
51.8 percent were recommended for a developmental writing course. Twenty-nine percent of all students 
were recommended to enroll in developmental courses in all three subjects. 
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In specifying this second-stage model, I used generic outcome Y, to refer to each of 
the eight outcomes described above. In cases where there were dichotomous outcomes at 
both the first and second stages, I adopted a bivariate probit model at both stages, 
estimating marginal effects evaluated at the sample mean to summarize the effects that 
enrollment in developmental education had on the probability that a student was enrolled in 
the second semester and/or the second year. For continuous outcomes, I fitted linear 
regression models using OLS methods. The parameter of interest in Equation 2 is Pi, 
representing the difference in outcome between treatment and control students, at the 
discontinuity, on average in the population. 


Research Question 2: Is participation in redesigned remedial courses more 
effective than participation in traditional remedial courses offered at similar 
institutions during the same time period with regard to the impact on subsequent 
student academic outcomes? 

I am also interested in whether the impact of enrollment in developmental math at 
the three sample institutions differed from the impact of enrollment in developmental math 
at similar types of institutions in Tennessee that offered traditional remedial courses during 
the same time period. A plausible hypothesis might be that, among students enrolled in two- 
year schools, enrollment in developmental math has a g reater impact on those attending 
colleges that implemented a redesign. Incorporating data from the other 16 colleges in 
Tennessee in the years 2008-09 and 2009-10 (for the post-reform cohorts) allowed me to 
answer my second research question. 

To answer this question, I again used a fuzzy regression discontinuity design, but 
this time I included data from all 1 9 publ ic TBR institutions in the analysis and 
incorporated in the statistical model a two-way interaction term between the instrumental 
variable, ASSIGN, and whether or not a student attended an institution that implemented a 
redesign, hereafter known as a reform institution. As before, in my first-stage model, I 
regressed whether a student enrolled in a remedial course on w hether the student was 
assigned to this course based on hi s or her position with respect to the cutoff on the ACT 
Math exam, as in Equation 1 , but in this model I also included a dummy predictor, INST, to 
identify those three institutions that implemented a redesigned developmental math 
curriculum (INST = 1) from those institutions that did not (INST = 0), in the years 2008-09 
and 2009-10. Thus, my first-stage model then becomes: 

DEV i =y a + y t ASSIGN, + y .SCORE, + y 3 (ASSIGN *SCORE) : + y 4 INST : 

(3) + y 5 (ASSIGN * INST), + y 6 (SCORE * INST), + y. ( SCORE * INST * ASSIGN), + y,Z, + S t 
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for the I th individual and where 8\ is the first-stage residual. Z is the same vector of 
covariates as in Equation 1. I again introduced the fitted probability of enrolling in 
developmental courses as the critical question predictor in the second-stage statistical model 
to estimate the causal effect of the take-up of remediation on outcome F, as follows: 

Y i =j3 a + p x DEV,+ p, SCORE, + pp ASSIGN *SCORE ), + PJNST, 

(4) + p, (j DEV * INST), + p 6 (ASSIGN * INST), + p i (SCORE * INST), 

+ PPSCORE * INST * ASSIGN), + fi 9 Z t + e, 

with the usual notation and a suitable correction to the standard errors of the estimates. 
Equation 4 contains two parameters of interest: Pi represents the causal effect of enrollment 
in remediation on the outcomes of interest within all four-year and two-year TBR colleges 
during the post-reform years (2008-09 and 2009-10) and p 5 provides an estimate of how 
the main effects captured in Pi differ as a result of being enrolled in one of the three reform 
institutions. 

Research Question 3: Is participation in redesigned remedial courses more 
effective than participation in the traditional model of remediation at the same 
institution prior to the implementation of these new courses? 

The results from my second research question provide a comparison of the impacts 
on outcomes of re designed remedial courses to traditional remedial courses using similar 
institutions across Tennessee in the same time period. This strategy, however, does not 
account for the fact that the three reform institutions may be different in unobservable ways 
from other seemingly similar institutions across the state. It may be that the three 
institutions that applied for and received funding to implement course redesigns already had 
a culture of innovation or an infrastructure to support institutional reforms. If this were the 
case, comparing these institutions to their peers in the same sector (four year vs. two year) 
would overestimate the effects of these redesign efforts. To address this concern, I used an 
IV estimation similar to that described above for Research Question 2; however, I restricted 
the sample to include only the three institutions that implemented a course redesign as with 
Research Question 1 , but I also included data from the two cohorts of s tudents entering 
these institutions prior to the redesign (2006-07 and 2007-08). Consequently, I included in 
the analysis a two-way interaction term between the instrumental variable, ASSIGN, and 
whether or not a student enrolled in their institution post-redesign (POST). 
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Determining the Bandwidth 

A critical component of any regression discontinuity analysis is the selection of a 
bandwidth around the cutoff score on the forcing variable within which data are included in 
the sample and the statistical models fitted. By selecting a s mailer bandwidth near the 
cutoff, one gains more confidence in the linearity of the outcome/forcing variable 
relationship that is driving the regression discontinuity projection. Alternatively, increasing 
the bandwidth increases the sample size and thus the statistical power of the analysis, but 
doing so also increases the sensitivity of the analysis to the functional form of the 
outcome/forcing variable relationship (Mumane & Willett, 2011). In what follows, I 
examined the sensitivity of my findings to bandwidth size empirically, using a cross- 
validation procedure developed by Imbens and Lemiuex (2008) to estimate the optimal 
bandwidth. I determined that the preferred bandwidth obtained using this procedure could 
not be greater than two points to the left of the cutoff, given the multi-tiered placement 
policy in Tennessee. Students scoring more than two points below the cutoff are 
traditionally no longer assigned to the first developmental course below college-level, but to 
the second or third. For this reason, I restricted the lower bound on the forcing variable to 
two ACT Math exam points, and used the procedure described above to estimate the 
optimal bandwidth above the cutoff score for all outcomes. Within each table of results, I 
present the optimal bandwidth for each outcome estimate. 


21 I assumed that the functional form to the left of the cutoff was linear, while I permitted the functional 
form to right of the cutoff to differ, according to the data. As a s ensitivity check, I re-estimated the 
findings for alternative functional forms to the right of the cutoff by including a quadratic term for my 
question predictor in Equations 2 a nd 4, and in all cases found that the estimates did not change 
significantly. While the size of the standard errors increased, these estimates did not differ from those 
obtained from the local linear specification. 
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4. Results 


In my analysis, I obtained consistent estimates of the causal impact of enrolling in 
developmental courses on subsequent outcomes for students on the margins of scoring 
above the cutoff on t he ACT Math exam that is used for pi acement into developmental 
math courses. As such, these results are externally valid only for students whose scores on 
the forcing variable fell close to the discontinuity in the placement scores. The results can 
therefore be generalized only to the population of students at the margins of passing. 

The results offer important evidence as to the potential power and influence of 
redesigning developmental math courses. Overall, I found that students who enrolled in 
developmental math courses had higher first-to-second semester persistence rates than did 
their peers enrolling in college-level courses after the first and second year; however, these 
effects tended to diminish over time. While there did appear to be some early gains in 
student persistence from the first to the second semester, these effects were no longer 
confirmed from the first to the second year. Interestingly, however, students exposed to 
redesigned developmental math courses had more positive outcomes than did their peers in 
non-redesign institutions, on average, during the same period and also when compared with 
students exposed to the previous version of traditional remediation within their institution in 
prior years. It appears that students enroll and persist at the same rate in the second year, but 
take more college-level courses as a result of the redesign process. 


Estimated Effects of Redesigned Remediation Courses, Fall 
2008-09 and 2009-10 Cohorts 

In Table 4, I present instrumental-variables estimates from the fitted two-stage 
least-squares models specified in Equations 1 and 2. These estimates summarize the causal 
effects of the redesigned developmental math courses on subsequent academic outcomes for 
students beginning in one of the three sample institutions in the fall of 2008-09 and 2009- 
10, at the margins of passing the placement test. This analysis most closely resembles prior 
research that sought to address research questions on c ollege remediation, as it simply 
compares the academic outcomes for students enrolling in a developmental math course 
with outcomes for similar students enrolled in a college-level math course. I restricted the 
sample to include only the three institutions that had first implemented redesigned 
developmental math courses in the fall of 2008, but pooled both the 2008-09 and 2009-10 
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Table 4 


Estimated Effects of Redesigned Remediation Courses, Fall 2008-09 and 2009-10 Cohorts 



FIRST SEMESTER 

FIRST YEAR 


SECOND YEAR 


Credits attempted 
but not completed 
in 1 st semester 

1 st to 2nd 
semester 
persistence 

College-level 
credits 
completed 
in 2nd semester 

Total credits 
completed 
after 1 year 

Still 

enrolled 
in year 2 

College-level 
credits 
completed 
in year 2 

College-level credits 
completed 
after 2 years 

Total credits 
completed 
after 2 years 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

REFORM INSTITUTIONS: POOLED DATA 








Assigned to Develop. 

-1.656** 

0.105** 

0.713 

1.876 

0.008 

1.321 

1.027 

2.040 

Algebra II 

(0.802) 

(0.047) 

(0.859) 

(1.695) 

(0.085) 

(2.276) 

(3.748) 

(3.270) 

Fitted value at cutoff 

3.40 

0.864 

11.59 

24.81 

0.647 

14.81 

33.56 

39.02 

Bandwidth 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 2 

-2 < x < 1 

-2 < x < 1 

-2 < x < 2 

Observations 

2392 

2392 

2392 

2392 

2135 

1804 

1804 

2135 

AUSTIN PEAY STATE UNIVERSITY 








Assigned to Develop. 

-1.477 

0.093+ 

0.688 

3.782 

-0.037 

0.866 

0.662 

3.497 

Algebra II 

(1.443) 

(0.060) 

(1.521) 

(2.998) 

(0.155) 

(2.276) 

(7.098) 

(6.106) 

Fitted value at cutoff 

3.08 

0.892 

12.41 

26.36 

0.697 

17.18 

37.92 

42.88 

Bandwidth 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 2 

-2 < x < 1 

-2 < x < 1 

-2 < x < 1 

Observations 

1309 

1309 

1309 

1309 

1137 

928 

928 

928 

CLEVELAND STATE COMMUNITY COLLEGE 








Assigned to Develop. 

-3.333* 

0.102 

0.614 

-0.823 

0.007 

1.234 

1.724 

-0.184 

Algebra II 

(1.762) 

(0.491) 

(1.839) 

(3.602) 

(0.186) 

(2.523) 

(2.305) 

(3.485) 

Fitted value at cutoff 

4.22 

0.813 

10.65 

23.02 

0.573 

12.15 

28.41 

34.31 

Bandwidth 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 4 

-2 < x < 4 

-2 < x < 3 

Observations 

459 

459 

459 

459 

459 

489 

489 

459 

JACKSON STATE COMMUNITY COLLEGE 








Assigned to Develop. 

-0.827 

0.148 

0.536 

1.744 

0.029 

2.531 

1.969 

1.688 

Algebra II 

(1.074) 

(0.847) 

(1.106) 

(2.272) 

(0.803) 

(2.748) 

(2.424) 

(2.007) 

Fitted value at cutoff 

3.38 

0.849 

10.71 

23.10 

0.605 

12.17 

28.89 

35.01 

Bandwidth 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

Observations 

624 

624 

624 

624 

624 

624 

624 

624 


NOTES: Robust standard errors are shown in parentheses. The pooled data sample includes all first-time, full-time students under the age of 21 with a reported ACT Math Score at Austin Peay State 
University, Cleveland State Community College, and Jackson State Community College in the fall of 2008 or the fall of 2009. For the four binary persistence outcomes, the marginal effects and 
standard deviations are reported. Control variables include gender, race/ethnicity, age, high school GPA, and a dummy variable for whether a student was assigned to a developmental/remedial 
reading or writing course. 

The bandwidth on either side of the cutoff is calculated individually for each outcome using the cross-validation procedure developed by Imbens and Lemiuex (2008). The statewide cutoff policy 
used to assign students to developmental math courses is used as an instrument for enrollment in developmental math. For students who dropped out, the number of credits is listed as the number of 
credits the student had completed when last enrolled. 

+p < .15. *p<.10. **p < .05. ***p < .01. 
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cohorts, as the redesigned courses were offered in subsequent semesters. - In the top row of 
Table 4, 1 present estimated effects pooled across all three institutions and in the subsequent 
rows break out the findings by individual institution. The outcomes are ordered 
chronologically, with first-semester outcomes in columns 1-2, first-year outcomes in 
columns 3 4, and second-year outcomes in columns 5-8. Additionally, I provide the fitted 
value of each outcome at the cutoff (when SCORE = 0) as a reference point for the size of 
the main effect found in the first row of each section of the table. I also list the optimal 
bandwidth for each outcome within each institution, as well as the number of observations 
within this bandwidth. 

An early measure of potential student academic difficulty can be gauged by the 
number of credits a student attempts in the first semester but does not complete. In column 
1 , I show the effects of enrollment in developmental math on the number of c redits 
attempted but not completed in the first semester. At Cleveland State, for e xample, the 
number of credits attempted but not completed for s tudents just above the cutoff for 
assignment to developmental math was 4.22 credits, which includes those students who 
completed all of the credits they attempted (and thus have a value of zero for this outcome). 
Among those enrolled in developmental math, however, students completed an average of 
3.3 credits more of their attempted credits (or an average of one course) than their peers 
assigned to college-level math. For Cleveland State, it appears that enrollment in 
developmental math had a positive effect on t he number of credits attempted but not 
completed in the first semester. I present these results graphically in Figure 3 for both the 
pooled reform institutions and for Cleveland State. In the pooled sample plot on the left, on 
average, students enrolling in developmental math completed 1.6 more of their attempted 
credits than did their peers who enrolled in college-level math. This gap is even wider 
among students at Cleveland State, shown on the right. For the fitted values to the right of 
the vertical cutoff on both graphs, I present both the linear specification and a quadratic 
specification to illustrate that the results do not differ by the choice in the functional form of 
the outcome/forcing variable relationship. 


22 Students attending one of the three sample institutions in the fall 2009-10 cohort would have been 
exposed to the same basic redesigned developmental mathematics courses as their peers beginning in the 
fall 2008. However, it may likely be the case that the delivery and instruction of these courses improved 
over the course of a year from the pilot year in 2008, in which case pooling both cohorts may result in a 
slight overestimation of the effects that would have otherwise been observed if examining only the pilot 
year. Arguably, given the many challenges that accompany implementing any pilot program, including 
the second cohort of students is more representative of the true effects of these resigned courses than 
simply focusing on the pilot year. 
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Figure 3 


Fitted Values Estimating Enrollment in Developmental Math on the Number of Credits 
Attempted but Not Completed After Two Years for Students Beginning in the Fall of 

2008-09 and 2009-10 (from Table 4) 


Credits Attempted But Not Completed (2008-09 & 2009-10 Cohorts) 

Pooled Reform Institutions Cleveland State Community College 




ACT Math Score Relative to Cutoff ACT Math Score Relative to Cutoff 


NOTES: The sample for the Pooled Reform Institutions includes all first-time, full-time students under 
the age of 21 with a reported ACT Math score at Austin Peay State University, Cleveland State 
Community College, and Jackson State Community College. Each circle represents the sample mean of 
the dependent variable for students with a given ACT Math score, with the vertical line representing the 
statewide cutoff for placement into college-level math. The dashed lines represent the fitted values for the 
outcome on the assignment to treatment variable by ACT Math score. Control variables include gender, 
race/ethnicity, age, high school GPA, and a dummy variable for whether a student was assigned to a 
developmental/remedial reading or writing course. The bandwidth on either side of the cutoff is estimated 
individually for each outcome using the cross-validation procedure developed by Imbens and Lemiuex 
(2008). 


The effects of the redesigned remediation on the dichotomous outcomes describing 
student persistence (columns 2 and 5 of Table 4) are estimated using a bivariate probit 
model, estimating marginal effects at the sample mean. Across the three institutions in the 
sample, for students at the margins of placing into a college-level math course, enrolling in 
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Developmental Algebra II increased the probability of persisting from the first to the second 
semester by 10.5 percentage points compared with students in college-level math, as shown 
in the first row of column 2. This estimate includes students who enrolled full time in the 
fall of their first semester but subsequently enrolled either full or part time in the following 
spring. The statistically significant effects on pe rsistence from the first to the second 
semester, however, do not hold up when examining the effects on persistence from the first 
to the second year (column 5), as none of t he coefficients on pe rsistence can be 
distinguished from zero, across either the pooled or the individual institutions, in the 
population. 

While one would expect that students assigned to developmental math courses 
would earn fewer college-level credits in the first semester, by the second semester one 
might speculate that these students would be enrolling in college-level courses at the same 
level as their peers who did not enroll in developmental math in the first semester. In 
column 3, 1 present estimates of the effect of enrollment in a redesigned remedial course on 
the number of college-level credits completed in the second semester only. Across all three 
institutions for the 2008-09 and 2009-10 cohorts, as well as individually within each 
institution, it appears that both treatment students (who enrolled in developmental math in 
the first semester) and control students (who enrolled in college-level math in the first 
semester) took a similar number of college-level credits in the second semester. The same is 
true for t he number of college-level credits completed in the second year, as well as 
cumulatively after two years (columns 6 and 7), although these estimates are not 
statistically significant by the end of year two. Similarly, students enrolled in developmental 
math did not accumulate any more or less total credits than did their peers enrolled in 
college-level math after the first or second year. 


Estimated Effects of Redesigned Remediation Courses Versus 
Traditional Remediation Courses at Other Two- and Four-Year 
Institutions, Fall 2008-09 and 2009-10 Cohorts 

Table 5 presents the effects of remediation for the three redesign institutions, but 
now relative to the other public institutions in the same sector. The top row in both the four- 
year and two-year panels of Table 5 displays the effect of assignment to developmental 
math across all four-year and two-year institutions in the state, respectively. Across all four- 
year colleges, students who enrolled in a developmental math course completed 0.5 fewer 
college-level credits in the second semester compared with their peers enrolling in college- 
level math, and this gap jumped to two credits after two years (both significant at the 10 
percent level). Column 8 suggests that students who enrolled in developmental math had 
nearly three fewer total credits by the end of two years than their peers enrolling in college- 
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Table 5 


Comparing Redesigned Remediation Courses with Traditional Remediation Courses at Other Two and Four-Year Institutions, 

Fall 2008-09 and 2009-10 Cohorts 



FIRST SEMESTER 

FIRST YEAR 


SECOND YEAR 


Credits 
attempted but 
not completed in 
1 st semester 

1st to 2nd 
semester 
persistence 

College-level 
credits 
completed 
in 2nd semester 

Total credits 
completed 
after 1 year 

Still 

enrolled 
in year 2 

College-level 
credits 
completed 
in year 2 

College-level 
credits 
completed 
after 2 years 

Total credits 
completed 
after 2 years 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

ALL FOUR YEAR COLLEGES (Redesigned Remediation = Austin Peay State University) 






Assigned to Develop. 

-0.578 

0.025 

-0.492* 

-1.146 

0.019 

-0.399 

-1.972* 

-2.873 

Alg. II ((30 

(0.471) 

(0.021) 

(0.185) 

(1.009) 

(0.036) 

(1.400) 

(2.317) 

(1.987) 

Redesigned Remediation 

-0.261 

-0.007 

0.026 

-0.488 

0.007 

-0.532 

-0.739 

-0.803 

(P 4 ) 

(0.174) 

(0.012) 

(0.179) 

(0.501) 

(0.017) 

(0.517) 

(0.856) 

(0.734) 

Assigned * Redesigned 

0.115 

-0.013 

-0.698* 

0.077 

-0.006 

1.145 

2.694* 

0.949 

Remediation (p 5 ) 

(0.517) 

(0.027) 

(0.331) 

(1.238) 

(0.038) 

(1.534) 

(1.539) 

(2.177) 

Fitted value at cutoff 

3.02 

0.904 

12.90 

27.26 

0.707 

17.94 

39.77 

44.69 

Bandwidth 

-2 < x < 3 

-2 < x < 2 

-2 < x < 3 

-2 < x < 1 

-2 < x < 1 

-2 < x < 3 

-2 < x < 3 

-2 < x < 2 

Observations 

10,996 

9,595 

10,996 

7,845 

7,845 

10,996 

10,996 

9,595 

H„: P, + Ps = 0 

0.540 

0.567 

0.095 

0.525 

0.833 

0.740 

0.086 

0.546 

ALL TWO YEAR COLLEGES (Redesigned Remediation = Cleveland State and Jackson State Community Colleges) 





Assigned to Develop. 

-0.378 

0.004 

-0.442 

-1.515* 

0.076* 

2.069* 

-0.612 

-0.612 

Alg. II (Pi) 

(0.453) 

(0.033) 

(0.493) 

(0.864) 

(0.043) 

(1.228) 

(1.754) 

(1.754) 

Redesigned Remediation 

0.147 

-0.005 

0.038 

0.203 

-0.064*** 

-1.552** 

-1.125 

-1.125 

(P 4 ) 

(0.218) 

(0.019) 

(0.255) 

(0.463) 

(0.023) 

(0.635) 

(0.842) 

(0.842) 

Assigned * Redesigned 

0.143 

-0.012 

-0.145 

0.466 

-0.014 

0.072 

-1.035 

-0.035 

Remediation (p 5 ) 

(0.378) 

(0.031) 

(0.424) 

(0.804) 

(0.038) 

(1.056) 

(1.462) 

(1.462) 

Fitted value at cutoff 

3.77 

0.821 

10.58 

22.55 

0.613 

13.07 

29.70 

34.95 

Bandwidth 

-2 < x < 3 

-2 < x < 3 

-2 < x < 2 

-2 < x < 3 

-2 < x < 3 

-2 < x < 2 

-2 < x < 2 

-2 < x < 3 

Observations 

9,163 

9,163 

8,379 

9,163 

9,163 

8,379 

8,379 

9,163 

H 0 :Pi + P 5 = 0 

0.613 

0.908 

0.252 

0.103 

0.258 

0.093 

0.089 

0.719 


NOTES: See notes for Table 5. Robust standard errors are shown in parentheses. The pooled data sample includes all first-time, full-time students under the age of 21 with a reported ACT Math Score at Austin 
Peay State University, Cleveland State Community College, and Jackson State Community College in the fall of 2008 or the fall of 2009. Redesigned remediation is equal to 1 when a student attends either 
Austin Peay University among the four-year institutions, or Cleveland State or Jackson State among the two-year institutions. 


+p<.15. *£><.10. **£> < .05. ***£><.01. 
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level math; however, these results are not statistically significant. Within the two-year 
colleges, students assigned to developmental math completed an average of two additional 
college-level credits in their second year compared with their peers in college-level math 
(column 6). 

The interaction term in Table 5 provides the effect for the redesign institutions 
compared with the other public four-and two-year colleges in the state during the time 
period post-reform (2008-09 and 2009-10). Relative to other four-year institutions, 
students in developmental math at Austin Peay completed 2.7 more college-level credits 
than their peers after two years. Given the details of the Austin Peay redesign, this is not 
surprising. Whereas other students who enrolled in developmental math at the other five 
four-year colleges did not receive college-level credits for these courses, students at Austin 
Peay were mainstreamed into college-level math classes and thus received college-level 
credit. Therefore, these students had more college-level credits after one and two years 
compared with students attending their peer four-year institutions. This result is shown 
graphically in Figure 4, with the gap in the number of c ollege-level credits between 
remedial and college-level students at all four-year colleges shrinking considerably for 
students attending Austin Peay State University. On Table 5, 1 also include the results from 
a general linear hypothesis (GLH) test to test if the sum of the (3j and (3s is zero. For the 
number of college-level credits completed after two years, the 0.722 difference in credits 
between Austin Peay and the other four-year colleges in the state is statistically different 
than zero, as shown by the last row of column 7. 

Within the two-year colleges, the effects at Cleveland State and Jackson State did 
not differ in most cases from the overall effects of remediation within the two-year colleges. 
For the number of total credits completed after one year, students who attended one of the 
two redesigned math courses completed an average of 0.5 more total credits compared with 
students at the other nine community colleges, who completed an average of 1.5 fe wer 
credits than did their peers enrolled in college-level math. Table 5 provides interesting 
results relative to the non-redesign institutions. These findings confirm that assignment to a 
newly designed remedial program had a causal effect on students’ subsequent academic 
outcomes among students at the margins of passing, and that these effects differ by 
institution. 
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Figure 4 


Effects of Enrollment in Developmental Math on the Number of College-Level Credits 
Completed After One Year, Comparing Austin Peay University with Other Four-Year 
Public Colleges, 2008-09 and 2009-10 Cohorts (from Table 5) 


College-Level Credits Completed After Two Years (2008-09 & 2009-10 Cohorts) 
All Four-Year Colleges Austin Peay State University 



ACT Math Score Relative to Cutoff 


ACT Math Score Relative to Cutoff 


Estimated Effects of Traditional Versus Redesigned 
Remediation Courses at the Redesign Institutions 

Using data for the two cohorts prior to the developmental math redesigns (fall 
2006-07 and fall 2007-08), Table 6 presents the effects of enrollment in developmental 
math on the academic outcomes for students at the three reform institutions, but also 
includes a cohort effect ( POST) to allow for a comparison with the more traditional 
developmental education model that was being used prior to the 2008-09 course redesigns. 
The outcomes in Table 6 are the same as in prior tables. 
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Table 6 


Estimated Effects of Traditional Versus Redesigned Remediation Courses at the Reform Institutions 



FIRST SEMESTER 

FIRST YEAR 


SECOND YEAR 


Credits 
attempted but 
not completed 
in 1st 
semester 

1 st to 2nd 
semester 
persistence 

College-level 
credits 
completed 
in 2nd semester 

Total credits 
completed 
after 1 year 

Still 

enrolled 
in year 2 

College-level 
credits 
completed 
in year 2 

College-level 
credits 
completed 
after 2 years 

Total credits 
completed 
after 2 years 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

REFORM INSTITUTIONS: POOLED DATA 







Assigned to 

0.481* 

0.038* 

-1.181 

-0.955 

-0.063 

-1.985 

-5.189* 

-3.919+ 

Develop. Alg. II 
(pi) 

(0.244) 

(0.018) 

(0.752) 

(1.370) 

(0.056) 

(1.772) 

(2.933) 

(2.630) 

Post Reform ((3 4 ) 

-0.182 

0.039** 

-0.094 

0.695 

-0.016 

-1.361* 

-1.684 

-1.330 

(0.293) 

(0.016) 

(0.335) 

(0.624) 

(0.026) 

(0.807) 

(1.337) 

(1.198) 

Assigned * Post 

-1.336* 

0.033* 

1.651* 

1.539 

0.041 

3.387* 

5.971* 

4.907+ 

Reform ((3 5 ) 

(0.732) 

(0.015) 

(0.937) 

(1.556) 

(0.062) 

(2.012) 

(3.332) 

(2.987) 

Fitted value at cutoff 

3.72 

0.850 

11.54 

24.37 

0.646 

15.30 

34.19 

39.32 

Bandwidth 

-2 < x < 3 

-2 < x < 2 

-2 < x < 2 

-2 < x < 3 

-2 < x < 1 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

Observations 

4,666 

4,168 

4,168 

4,666 

3,510 

4,666 

4,666 

4,666 

H 0 : Pi + Ps = 0 

0.094 

0.079 

0.568 

0.702 

0.918 

0.147 

0.811 

0.086 

AUSTIN PEAY STATE UNIVERSITY 








Assigned to 

1.559 

-0.043* 

-2.216* 

-3.252 

-0.165* 

-6.211* 

-6.189** 

-4.832** 

Develop. Alg. II 

(p,i 

(1.192) 

(0.021) 

(1.273) 

(2.051) 

(0.087) 

(3.339) 

(2.960) 

(1.922) 

Post Reform ((3 4 ) 

0.060 

0.037** 

-0.039 

0.574 

-0.001 

-1.371 

-1.768 

-2.335 

(0.416) 

(0.018) 

(0.444) 

(0.805) 

(0.034) 

(1.165) 

(1.941) 

(1.718) 

Assigned * Post 

-2.305 

0.076+ 

2.609 

4.071 

0.147* 

6.063 

2.475* 

5.781* 

Reform (P s ) 

(1.783) 

(0.044) 

(1.904) 

(2.523) 

(0.076) 

(4.995) 

(1.318) 

(2.963) 

Fitted value at cutoff 

3.34 

0.883 

12.35 

25.99 

0.690 

17.57 

38.50 

43.34 

Bandwidth 

-2 < x < 1 

-2 < x < 3 

-2 < x < 1 

-2 < x < 3 

-2 < x < 1 

-2 < x < 1 

-2 < x < 1 

-2 < x < 1 

Observations 

1,762 

2,500 

1,762 

2,500 

1,762 

1,762 

1,762 

1,762 

X 

©. 

+ 

XD 

II 

O 

0.619 

0.100 

0.806 

0.765 

0.403 

0.972 

0.119 

0.178 
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FIRST SEMESTER 

FIRST YEAR 


SECOND YEAR 


Credits 
attempted but 
not completed 
in 1st 
semester 

1 st to 2nd 
semester 
persistence 

College-level 
credits 
completed 
in 2nd semester 

Total credits 
completed 
after 1 year 

Still 

enrolled 
in year 2 

Col lege- level 
credits 
completed 
in year 2 

College-level 
credits 
completed 
after 2 years 

Total credits 
completed 
after 2 years 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(V) 

(8) 

CLEVELAND STATE COMMUNITY COLLEGE 







Assigned to 

-0.407 

0.154* 

0.513 

0.549 

0.058 

-3.909* 

-1.431 

-1.431 

Develop. Alg II 
(Pi) 

(1.515) 

(0.084) 

(1.785) 

(3.280) 

(0.118) 

(2.197) 

(2.460) 

(2.460) 


0.465 

0.030 

-0.496 

-0.447 

-0.025 

-3.826* 

-4.104 

-4.104 

rost Ketorm (p 4 j 

(0.720) 

(0.036) 

(0.830) 

(1.559) 

(0.048) 

(1.995) 

(3.003) 

(3.003) 

Assigned * Post 

-2.084 

-0.357** 

0.382 

-0.651 

0.031 

6.213+ 

1.979 

1.979 

Reform ((3 5 ) 

(1.607) 

(0.167) 

(2.099) 

(3.478) 

(0.141) 

(3.850) 

(2.597) 

(2.597) 

Fitted value at cutoff 

4.06 

0.821 

11.23 

24.02 

0.614 

13.82 

31.626 

36.92 

Bandwidth 

-2 < x < 3 

-2 < x < 2 

-2 < x < 2 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 2 

-2 < x < 2 

Observations 

886 

808 

808 

886 

886 

886 

808 

808 

©_ 

+ 

XD 

II 

O 

0.089 

0.201 

0.597 

0.975 

0.616 

0.140 

0.929 

0.929 

JACKSON STATE COMMUNITY COLLEGE 







Assigned to 

-0.685 

0.113* 

0.407 

1.170 

0.016 

2.928 

2.102 

3.217 

Develop. Alg II 
(pi) 

(1.020) 

(0.067) 

(1.073) 

(2.139) 

(0.087) 

(2.581) 

(4.160) 

(3.854) 

Post Reform ((3 4 ) 

-0.873 

0.041 

0.279 

1.740 

-0.033 

-0.542 

-0.087 

0.939 

(0.570) 

(0.033) 

(0.600) 

(1.196) 

(0.042) 

(1.443) 

(2.325) 

(2.154) 

Assigned * Post 

0.449 

-0.055 

-0.893 

-1.082 

0.079 

-1.198 

-2.534 

-3.708 

Reform ((3 5 ) 

(1.195) 

(0.104) 

(1.257) 

(2.508) 

(0.122) 

(3.026) 

(4.876) 

(4.518) 

Fitted value at cutoff 

4.18 

0.807 

10.18 

21.49 

0.584 

12.01 

27.79 

33.38 

Bandwidth 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

-2 < x < 3 

Observations 

1,280 

1,280 

1,280 

1,280 

1,280 

1,280 

1,280 

1,280 

©. 

XD 

+ 

XD 

II 

O 

0.815 

0.139 

0.647 

0.967 

0.956 

0.498 

0.916 

0.897 


NOTES: See notes for Table 5. Robust standard errors are shown in parentheses. The pooled data sample includes all first-time, full-time students under the age of 21 with a 
reported ACT Math Score at Austin Peay State University, Cleveland State Community College, and Jackson State Community College. 


POST = 1 for those cohorts beginning in 2008-09 and 2009-10 and POST = 0 for those cohorts beginning in 2006-07 and 2007-08. 
+p<.15. *p < .10. **p < .05. ***p<.01. 
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The top row of each section of Table 6 reports the cumulative effect of assignment 
to developmental math across all four cohorts in the data. Generally across the three pooled 
institutions, it appears that participation in a remedial math course had negative effects on 
the number of credits attempted but not completed and on the number of college-level 
credits a student had completed by the end of the first and second years, with the greatest 
deficit in the number of college-level credits (six by the end of the second year) observed at 
Austin Peay State University. The more interesting results in this table, however, are not the 
basic comparisons of students in developmental math with those in college-level math. The 
interaction term of Table 6 (|U) illustrates the comparison of redesigned courses with the 
prior traditional version of developmental courses. The coefficient on the interaction term 
provides the estimate of the impacts of enrollment in a redesigned developmental math 
course compared with enrollment in a traditional developmental math course for students 
within the same institution, pre- and post-redesign. For example, for students at Austin Peay 
University between 2006-07 and 2009-10, the overall effect of enrollment in a 
developmental math course led to a 4.3 percentage point increase in the likelihood of being 
enrolled in the second semester compared with students assigned to a college-level course. 
For students who entered in the latter two cohorts (those with redesigned developmental 
courses), however, the likelihood ofbe ing enrolled in the second semester was 7.6 
percentage points higher than it was previously. The overall effect on student persistence is 
the sum of the coefficient on the main effect plus the coefficient on the interaction term. 

At Cleveland State, the redesigned courses led to more college-level credits after 
two years. Given that Cleveland State adopted an acceleration model by which students 
could move to the next course as soon as they had completed the prior, these are 
encouraging results. One would think that those who refused to take remediation might be 
the better students; thus one might expect, all else equal, that remediation would have worse 
effects. That does not appear to be the case here, however. Overall, the redesign appears to 
have been an improvement from the prior course offerings at two of the three schools. For 
students at Jackson State, it appears that the redesigned courses did not differ in their impact 
compared with the traditional developmental courses offered in the earlier years, as we see 
no statistically significant effects on the coefficients in Table 6. At Jackson State, students 
in the redesigned math courses proceeded through the required modules at their own pace. 
When one semester ended and another began, students could resume work on any modules 
not completed, and therefore the traditional definitions of semester and course credit are 
much looser in this case. 
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5. Conclusion and Implications 


The redesigned remedial courses that took effect in Tennessee in 2008 provided a 
natural experiment for estimating the causal effects of different models of instruction and 
delivery for these courses on student persistence and course-taking behavior. In this study, I 
investigated not only whether remediation helped to improve students’ college outcomes, 
but also what role course design played in determining student success. 

The results of the study offer important evidence as to the potential power and 
influence of re designing developmental math courses. Overall, I found that students in 
developmental math courses completed more of their attempted credits and had higher rates 
of persistence from the first to the second semester; however, these effects tended to 
disappear by the second year. Interestingly, students exposed to redesigned developmental 
math courses had more positive outcomes than did their peers in non-redesign institutions 
during the same period, and also when compared with students exposed to the previous 
version of traditional remediation within their institution in prior years. Students appeared 
to benefit the most from redesigned courses at Austin Peay State University and Cleveland 
State Community College. 

At Austin Peay State University, the four-year college, I found that the elimination 
of developmental math courses had a positive impact on early student persistence. I also 
found an increase in the number of college-level credits accumulated by the end of the first 
and second years when compared with other four-year institutions or with more traditional 
methods of offering remediation. This was the only institution of the three that implemented 
mainstreaming, or the elimination of developmental math courses in lieu of placing students 
into college-level courses with special outside supports. With strong positive results relative 
to their prior methods of teaching remedial math, the case could be made for endorsing 
mainstreaming remedial education more broadly. However, it is important to remember that 
we cannot separate out the particular culture of the institution from the impact of t he 
redesign model. It may very well be the case that the institution and its students and 
educators aided in the adoption of this mainstreaming approach in such a way that the 
effects seen here would not translate universally to all institutions. Furthermore, this result 
applies only to students at the margins of needing math remediation and therefore cannot be 
extrapolated to students far down the ability distribution, in which case enrollment in a 
college-level math course could potentially lead to more harmful academic effects than 
positive ones. These caveats aside, however, the results do seem to provide a strong 
endorsement of the notion that far too many students are placed in developmental courses at 
this institution. Given that students on the margins of needing developmental math were 
allowed to enroll directly in college-level math and that these students showed strong gains 
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when compared with prior years or with comparable institutions, it would be worthwhile to 
consider lowering the cutoff to avoid overplacement in remedial courses. A more accurate 
placement system could lead to a reduction in the number of di fficulties that arise from 
improper placement. 

A common component of each of the redesign efforts was the use of technology in 
the assessment and delivery of the course material. The study’s results support the notion 
that, at the very least, the adoption of such technologies does not have a negative effect on 
student outcomes, even in institutions where such new developments represent a dramatic 
change from prior methods of of fering such courses. Increased modularization and 
assessments should enable institutions to continue to customize remediation to best suit 
individual student needs. Additionally, in progress reports provided from the three 
institutions in the study, all three indicated that they were witnessing a more positive 
attitude toward the use of t echnology and the possibilities it created for fut ure redesign 
efforts (NCAT, 2009). 

Colleges and universities should also focus their efforts on helping students 
assigned to remedial courses to make continued progress toward their degrees. While taking 
remedial courses may not have large effects on short-term persistence, as shown in this 
study, it does significantly affect the number of c ollege-level credits a s tudent has 
completed by the end of the second year. Credit accumulation may be one reason why some 
students in need of re mediation obtain degrees at rates lower than their peers. For this 
reason, it is important to consider ways in which students can complete their remedial 
requirements, yet not be deterred from taking additional courses. The acceleration model 
implemented by Cleveland State Community College and the modules developed by 
Jackson State Community College are two examples of i nstitutional efforts to encourage 
credit accumulation among their students. As students complete one course and/or module, 
they are encouraged to begin the next step in the sequence immediately, which allows 
students the opportunity to move on t o their college-level courses more quickly. These 
efforts may prove to have positive long-term effects on s tudent persistence and degree 
attainment if college-level credit accumulation is indeed one of the key barriers to student 
success. 

Developmental education is a particular concern in Tennessee today. In 2010 the 
state adopted the Complete College Tennessee Act, which will take effect in the fall of 
2012. Under this act, all four-year colleges and universities will be prohibited from offering 
remedial education courses. Students will instead be able to co-enroll in four-year colleges 
and community colleges until they complete their remedial instruction. In addition, this act 
requires the development of a strategic plan for higher education and the development of a 
performance funding model that will likely include performance measures related to 
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remedial education (Senate Bill 7006/House Bill 7008). It is too early to know what the 
long-term effects of t hese redesigned courses will be; however, the adoption of the 
Complete College Tennessee Act will ensure that all the community colleges in the state 
will begin to focus on the adoption of newly designed programs of their own. Given that the 
largest positive findings in this study were from the four-year institution, policymakers in 
Tennessee are encouraged to pay close attention to the ways in which eliminating 
developmental education affects students at these institutions. 

The redesigns in Tennessee also raise awareness that remediation efforts need not 
focus solely on t he skills students did not learn in the past, but can instead attempt to 
identify and provide the skills students will need for a future career or academic major. 
Traditionally, developmental education has been intended to address whatever skills may 
have been missed in high school (Education Commission of the States, 2012). The ways in 
which these three redesign efforts attempted to more closely identify the areas in which 
students most need improvement helps to redefine developmental education more as an 
academic support than as a curricular burden. Future redesign efforts should consider 
continuing this focus on differentiated delivery based on student skill and placement level 
as more institutions look to customize instruction to address specific student deficiencies. 

Further research in this area ought to include a deeper look at student grades, 
particularly in college-level math courses. Ideally the research needs to move beyond a 
study of the basic effects of developmental education; it should delve into the classroom to 
discover what is really going on in these courses with respect to teaching and learning and 
to examine how faculty members can best adapt these new innovations. Rather than 
focusing on basic credit accumulation, administrators, policymakers, and students are 
increasingly interested in what students need to know to be successful in college-level 
courses and what skills they most need for t he contemporary workplace. Scaling up 
successful programs is a continued challenge, particularly given evidence that 
developmental courses may affect students differently. Further work in Tennessee will 
explore this heterogeneity among subgroups to determine whether redesigned remedial 
courses are helping or hurting students differently. By thinking more creatively about how 
to respond to a variety of learning abilities, it may be possible to design courses that work 
well with a diverse array of students and institutions, which may ultimately lead to better 
learning for all. 
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Appendix 


Table A.l 

Summarizing Course Redesign at Each Campus 


Institution 

Remedial 

Courses 

Redesigned 

Redesign Model 

Details of Redesign 

Approx. Savings to 
Institution 

Austin Peay 
State 

University 

Elementary 
Algebra and 
Intennediate 
Algebra 

“Mainstreaming” 

Eliminate the 
developmental 
courses, which carry 
no university credit 

Enhanced sections of t he two core college-level courses, Fundamentals of 
Math and Elements of Statistics, will be created for students whose test scores 
place them in developmental math. These college-level courses will not change 
in content but will be linked to Structured Learning Assistance (SLA) 
workshops. Only the deficiencies which are deemed necessary for success in 
the core math course will be addressed during the workshops. 

Decrease instructional costs 
from $402,804 to $193,556. 

Cleveland State 

Community 

College 

Sequence of 3 
developmental 
math courses: 
Basic Math, 
Elementary 
Algebra and 
Intennediate 
Algebra 

“Acceleration” 

Students who 
complete a 
developmental math 
course before the end 
of the tenn will be 
allowed to begin the 
next developmental 
course immediately 

Each course consists of 10-12 modules. Students will meet one hour in class 
and two hours in a large computer lab. The one-hour class meetings will be 
held in small computer labs to allow students to work online; instructors will 
provide individual student assistance and review student progress. The large 
computer lab will be available 54 hours per week to allow students to work at 
their convenience. Course material will be organized into modules, which 
students will complete at the rate of one or more each week. All homework and 
testing will be done online. 

The traditional cost-per- 
student ranging from $236- 
$208 in the three courses 
will decrease to a range of 
$184 to $167, or 19%. 

Jackson State 

Community 

College 

Three 

developmental 
math courses 

“Modules” 

Combine three 
developmental studies 
math courses into one 
course broken up into 
12 modules 

A pretest on an established set of competencies will determine what concepts 
students will be required to master for their majors. Following this assessment, 
each student will receive an individualized learning contract based on 
academic background, learning preferences, identified gaps and educational 
goals that will provide a path to achieving the desired learning outcomes. 
Students will be required to master only the concept deficiencies determined 
by a pretest and those that are relevant to their career goals. Modules 1-3 
replaced Basic Arithmetic, Modules 4-7 replaced Developmental Algebra I 
and Modules 8-12 replaced Developmental Algebra II. 

The redesigned course will 
reduce the cost-per-student 
from $177 to $141, or 20%. 

Chattanooga 

State 

Community 
College ( not 
successfully 
implemented in 
fall 2008 ) 

Three 

developmental 
math courses 

Students will spend 2 
hours in class and 2 
hours in a computer 
lab staffed by faculty 
and professional tutors 
each week 

Students in this active learning environment will be able to progress at their 
own rate, receiving immediate feedback from the software and one-on-one 
assistance in the lab. When the redesign is fully implemented, students who 
fail a module will be able to pick up where they left off and not have to repeat 
the entire course. Students also may take challenge tests for module placement. 

Reducing the number of 
sections from 1 62 to 40 
annually will reduce cost- 
per-student from $191 to 
$ 1 64 in the redesign (annual 
savings, $10,000). 


SOURCE: Compiled from the National Center for Academic Transformation (http://www.thencat.org/) 
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